fix: validate cached ONNX before reuse to prevent empty-graph TensorRT build crash#16
Merged
Merged
Conversation
…T build crash An interrupted/failed ONNX export leaves a syntactically-valid-but-empty protobuf at onnx_path. The export cache check was existence-only, so the empty file was reused, feeding a 0-node/opset-less graph into polygraphy fold_constants -> ORT shape inference bails on opset<7 -> returns None -> get_num_nodes(None) crashes as "'NoneType' object has no attribute 'graph'". builder.py: add _onnx_cache_valid() (size>0, >=1 node, opset>=7, loaded structure-only) and gate the export cache-hit on it; delete + re-export a bad cache entry. models.py: fail fast in BaseModel.optimize() on a 0-node graph. Applies to every engine type (UNet/VAE enc+dec/ControlNet/safety). Co-Authored-By: Claude Sonnet 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exception: Acceleration has failed: 'NoneType' object has no attribute 'graph'onnx_path. The export cache check inEngineBuilder.build()was existence-only (os.path.exists(onnx_path)), so the corrupt file was silently reused instead of being re-exported.Optimizer.fold_constants()→ polygraphyfold_constants, whose ORT symbolic shape inference refuses opset < 7 and returnsNone; polygraphy then callsget_num_nodes(None), raising the opaqueNoneTypeerror deep in a third-party dependency.Found cached model: ...vae_decoder.engine.onnximmediately followed byVAE decoder: original .. 0 nodes, 0 tensors, 0 inputs, 0 outputs.Changes
src/streamdiffusion/acceleration/tensorrt/builder.py: adds_onnx_cache_valid(path)— checks file size > 0, ≥1 graph node, opset ≥ 7 (loaded structure-only, no external weight data, so it's cheap even for >2GB SDXL models). The export cache-hit check now requires this in addition toos.path.exists; on a stale/corrupt hit it logs a warning, deletes the bad file, and falls through to re-export.src/streamdiffusion/acceleration/tensorrt/models/models.py:BaseModel.optimize()now fails fast with a clearRuntimeErrorif the incoming graph has 0 nodes, instead of letting it reach polygraphy and surface as an unreadableNoneTypecrash.compile_vae_encoder/decoder/safety_checker/unet/controlnet) route throughEngineBuilder.build(), and every model subclassesBaseModel.Test plan
ast.parsesyntax checkcompile_*entry points route throughEngineBuilder.build()where the new guard lives.onnxto an empty-but-valid graph and confirm the build now logsCached ONNX invalid, re-exportingand completes successfully (not runnable in this dev environment — no GPU/TensorRT here)🤖 Generated with Claude Code